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ABSTRACT 

In a study in 1996, the Consortium on Chicago School Research 
found that many classroom assignments in writing and mathematics made only 
modest academic demands on high school students. In this study, the nature of 
assignments in schools participating in the Annenberg Challenge reform effort 
in 1998-1999 was studied, to determine whether teachers were assigning more 
demanding tasks. The analysis was based on 349 assignments from 74 teachers 
in 1997, 953 assignments from 116 teachers in 1998, and 715 assignments from 
87 teachers in 1999. Each summer following the year in which the assignments 
were collected, a team of 14 to 20 teachers from other Chicago public schools 
applied scoring rubrics to assess the authenticity of the intellectual work 
demanded in these assignments. After each assignment had been scored, the 
numerical scores were analyzed using a many-facet Rasch measurement model to 
create separate scales for each grade and subject, and then to divide scores, 
by degree of challenge. The quality of classroom assignments in the field 
sample of Annenberg Challenge schools improved between 1997 and 1999. The 
level of authenticity in those assignments described as challenging had 
clearly risen in the period, but for those described as typical,, the results 
were more mixed. While these improvements are encouraging, the overall level 
of challenge ini mathematics assignments remains quite low. More than 80% of 
the sixth and eighth grade assignments in 1999 provided only minimal or no 
challenge. Writing assignments showed more evidence of challenge. A round of 
data collection in 2000-2001 will provide a more definitive basis for 
conclusions about school improvement. (Contains 4 figures, 2 tables, 9 
endnotes, and 10 references.) (SLD) 
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Background 

T he research reported here has been conducted by the Consortium 
on Chicago School Research for the Chicago Annenberg Challenge. 

The Challenge seeks to advance the educational opportunities af- 
forded Chicago’s children by supporting 45 networks of schools that work 
together with external partners to promote more ambitious intellectual work 
for all students. These efforts seek both to improve students’ basic skills in 
reading, mathematics, and writing, and to assure that all students have op- 
portunities to engage in academic work at a high level. 

Researchers at the Consortium on Chicago School Research are examin- 
ing academic progress in Annenberg Challenge schools in two ways. First, 
for all Annenberg Challenge schools we are undertaking trend analyses of 
student learning gains on basic skills, as measured by standardized tests. 1 
Second, we are complementing these test score analyses with an in-depth 
longitudinal study of the quality of intellectual work occurring in a sample of 
Annenberg Challenge schools. 

We view this two-pronged strategy as essential to developing a full picture 
of academic improvements in Annenberg Challenge schools. Trends on the 
Iowa Tests of Basic Skills (ITBS) have been improving systemwide since the 
early 1990s, with especially rapid gains in recent years. As the Chicago Public 
Schools have begun to use ITBS test score results for high-stakes account- 
ability — to intervene and reconstitute schools, to require students to attend 
summer school and even be retained in a grade, and as part of principals’ 
annual personnel evaluations — some reformers worry that these test score 
trends might be misleading. Incentives exist to “improve the scores” in ways 
that might have little to do with actual improvement in instruction. In our 
view, the gathering of actual classroom assignments and the analysis of the 
intellectual demands embedded in them offers a good complement to the 
test score data. These assignments — what teachers ask students to do on a 
day-to-day basis — are direct indicators of the quality of instruction. If 
instruction is genuinely improving, it should show up in changes in teach- 
ers’ assignments. 



In our view, the 
gathering of actual 
classroom assign- 
ments and the 
analysis of the 
intellectual de- 
mands embedded 
in them offers a 
good complement 
to the test score 
data. These assign- 
ments — what 
teachers ask stu- 
dents to do on a 
day-to-day basis — 
are direct indica- 
tors of the quality 
of instruction. 
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Has the Quality of Classroom Assignments Improved in 
Annenberg Challenge Schools? 



T his is the second Consortium report on the qual- 
ity of academic work in Chicago classrooms. 
Previously, in “The Quality of Intellectual Work 
in Chicago Schools: A Baseline Report” (Newmann, 
Lopez, and Bryk, 1998), we examined both classroom 
assignments from the first year of the study (1996-1997) 
and the student work performed on them. (The project 
will gather a final sample of assignments and student 
work in 2000-2001.) We found that the typical class- 
room assignment in both writing and mathematics made 
only modest academic demands on students. Generally, 
assignments consisted of simple drill and fill-in-the-blank 
practice exercises that rarely moved beyond rudimen- 
tary basic skills. However on a more encouraging note, 
we found that when teachers did assign more demand- 
ing classroom tasks, most students were able to com- 
plete them and demonstrate more complex intellectual 
performance. This brief asks whether we were seeing 
more of these kinds of assignments in 1998-1999 than 
when the Annenberg Challenge began in 1996. 

A Framework of Authentic Achievement as 
the Basis for Evaluating the Intellectual 
Demands of Classroom Work 

To guide our analysis of the intellectual demands em- 
bedded in classroom instruction, the Chicago 
Annenberg Research Project adopted the analytic 
framework of “authentic intellectual work” originally 
developed by the federally funded National Research 
Center on School Organization and Restructuring. 2 
In brief, authentic classroom instruction makes de- 
mands on students: 

• To apply basic knowledge and skills to solve new 
real world problems 

• To reach adequate solutions to these problems by 
organizing, interpreting, evaluating, and synthesiz- 
ing information 



• To ground their solutions in solid information, 
concepts, and principles from the academic disci- 
plines; and 

• To communicate effectively to others their conclusions. 

These intellectual skills are essential for success in the 
increasingly complex contemporary workplace where 
even entry level workers are asked to use knowledge to 
solve problems rather than just mechanically apply pre- 
viously learned facts and procedures. The rationale for 
this framework and greater detail on these key ideas are 
detailed in our first report. 3 

Collecting and Analyzing 
Classroom Assignments 

The Chicago Annenberg Research Project began collection 
of classroom assignments in 1 996- 1997 with 1 2 Annenberg 
elementary schools, expanding to 18 schools in 1997-1998, 
and 16 schools in 1998-1999. These schools represent a 
good cross-section of both Annenberg Challenge sites and 
the larger Chicago public school system. (A comparison of 
demographics in CPS, Annenberg Challenge, and sample 
schools is in Appendix A.) Within this sample of schools, 
we collected writing and mathematics assignments in 
grades three, six, and eight in order to span the typi- 
cal range of Chicago elementary school classrooms. 
These classes were selected because at the outset of the 
study these were the target grades for the statewide Illinois 
Goals Assessment Program (IG AP) . Data from these grades 
would allow us to link classroom assignments to student 
performance on state tests of reading, writing, and math- 
ematics as well as ITBS scores in reading and math. These 
results are presented in a separate report. 4 

Two teachers in each participating sample school, 
from grades three, six, and eight were asked to submit 
both typical assignments and challenging assignments 
in writing and math. Project researchers defined typical 
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assignments as reflecting the daily work occurring in the 
course of a regular school week. In contrast, challenging 
assignments were defined as those that the teacher be- 
lieved would provide the best indicators of how well stu- 
dents understood the subject at a high level. The teachers 
were asked to provide four typical assignments per year 
and two challenging assign- 
ments per year, for a total 
of six assignments. 5 The ac- 
tual number of assignments 
collected from teachers in 
a given year ranged from 
one to six, with most teach- 
ers providing four or five as- 
signments in each subject 
area. For the analysis re- 
ported here, we examined 
a total of 349 assignments 
from 74 teachers in 1997, 

953 assignments from 116 teachers in 1998, and 715 
assignments from 87 teachers in 1999.* See Appen- 
dix A for further details. 

Each summer following the school year that assignments 
were collected, a group of 14-20 teachers from other Chi- 
cago public schools was recruited to apply scoring rubrics 
that assess the authenticity of intellectual work demanded 
by the assignments. Teachers rated each assignment against 
the three standards of intellectual challenge: construction 
of knowledge, written communication, and connection 
to students’ lives. 6 The standard for construction of 
knowledge measured the extent to which the assignment 
asked students to interpret, analyze, or evaluate infor- 
mation, rather than simply reproduce facts or procedures. 
Written communication examined whether the task 
required students to draw conclusions, and support 
and elaborate them through extended writing. The 
standard for connection to students’ lives looked at 
the degree to which the assignment asked students to 
connect the topic or problem to their lives and whether 
the task resembled a problem that they might encoun- 
ter in daily life beyond school. 



Each summer we formed six teams of teacher raters, 
one for each subject matter/grade level combination (i.e., 
third grade writing, third grade math, etc.). Each team 
member was randomly assigned to score assignments, 
one standard at a time. As a result, each assignment was 
typically reviewed by three different raters, one for each 

standard. In order to con- 
trol for potential rater bias, 
a random sub-sample of as- 
signments was also scored 
by a second rater. The in- 
formation gained by this 
rescoring process allowed us 
to accurately assess and ad- 
just each assignment score 
for the differential effects 
associated with individual 
raters. 7 

After each assignment 
had been scored on the three standards, the numerical 
scores were analyzed to create an overall measure of in- 
tellectual quality. We used a many-facet Rasch measure- 
ment (MFRM) model to create separate scales for each 
grade and subject. The MFRM analysis also statistically 
adjusted the original scores for differences in the sever- 
ity of scorers and differences among the three stan- 
dards. 8 We then transformed these measures to a ten-point 
scale, with ten being the highest and zero being the lowest 
score. To provide a more substantive standards- based in- 
terpretation for these data, we also divided the distribu- 
tion of adjusted scores into four categories: extensive 
challenge, moderate challenge, minimal challenge, and 
no challenge. 

A sample of assignments scored in 1997 and 1998 was 
also scored in 1999. This allowed us to place assignment 
scores (within a grade and subject) from different years 
on a common scale that adjusted for differences over 
time in the relative severity of scorers and standards. All 
results reported here are in terms of the scales established 
as part of the ratings for the 1999 assignments. 9 



These [authentic] intellectual skills 
are essential for success in the increas- 
ingly complex contemporary work- 
place where even entry level workers 
are asked to use knowledge to solve 
problems rather than just mechani- 
cally apply previously learned facts 
and procedures. 



* As a shorthand, 1997 refers to the 1996-1997 academic year. The same is true for 1998 and 1999 respectively. 
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Examples of Sixth Grade Assignments 



High Scoring Writing Assignment 

Write a paper persuading someone to do something. Pick 
any topic that you feel strongly about, convince the reader 
to agree with your belief, and convince the reader to take a 
specific action on this belief. 

Commentary 

In this high scoring assignment, demands for construction 
of knowledge are evident because students have to select 
information and organize it into convincing arguments. By 
asking students to convince others to believe and act in a 
certain way, the task entails strong demands that students 
support their views with reasons or other evidence, which 
calls for elaborated written communication. Finally, the in- 
tellectual challenge is connected to students’ lives because 
they are to write on something they consider to be person- 
ally important. 



Low Scoring Writing Assignment 

Identify the part of speech of each underlined word below. 
All eight parts of speech — nouns, pronouns, verbs, adjec- 
tives, adverbs, prepositions, conjunctions, and interjec- 
tions — are included in this exercise. 

1) My room is arranged for comfort and efficiency. 

2) As you enter, you will find a wooden table on the left. 

3) I write and type. 

4) There is a book shelf near the table. 

5) On this book shelf, I keep both my pencils and pa- 
per supplies. 

6) [spend many hours in this room. 

7) I often read or write there during the evening. 

Commentary 

This assignment requires no construction of knowledge or 
elaborated communication, and does not pose a question 
or problem clearly connected to students’ lives. Instead it 
asks students to recall one-word responses, based on 
memorization or definitions of parts of speech. 



High Scoring Mathematics Assignment Low Scoring Mathematics Assignment 



Pick a stock. You have $10,000 to invest. Calculate how 
many shares you can buy at the current price. Every week 
for the next 10 weeks you will check in the newspaper 
whether your stock has gone up or down. You will chart 
the progress of your stock on the bulletin board. The chart 
is organized to record prices in 1/4 points, but the news- 
paper reports the prices in 1/16 points, so you will need 
to convert. 

At the end of the 10 weeks, determine whether you 
have made a profit or loss and predict what you think your 
stock will do based on the results of the chart. Decide 
whether you will buy more or sell your stock. At this point 
you will give an oral report on your stock, what happened to 
it, and what you decided to do. 

Commentary 

This assignment scores high on construction of knowledge 
because each week students must decide how to repre- 
sent the current price of the stock on a chart different from 
that which appears in the newspaper. They also have the 
opportunity to draw conclusions about their current profits 
or losses, which involves deciding what numbers to add, 
subtract, and multiply in order to compute them. The 
assignment’s demands for charting the stock requires some 
written mathematical communication, and one would as- 
sume that preparation for the oral report would entail some 
elaborated mathematical communication. By focusing on 
mathematics related to a stock that students choose to 
"own,” the assignment draws connections to mathematics 
and students’ lives beyond school. 



Name 


Adding and Subtracting Fractions and Mixed 


Numbers: Common Denominators 


Add or subtract. Reduce if possible. 




1. 2 2. 3 


3. 3 


4. 1 


3 8 


10 


4 


+ 1 +1 


+ 1 


+ 3 


3 8 


10 


4 


5. 7 6. 4 


7. 9 


8. 5 


8 5 


10 


6 


- 5 - 2 


- 3 


- 3 


8 5 


10 


6 





Commentary 

This assignment requires no construction of knowledge to 
address a mathematical problem, no extended writing to 
explain mathematical conclusions, and it does not pose a 
mathematical problem connected to students’ lives. Instead, 
it asks students only to fill in numerical answers to prob- 
lems on addition, subtraction, and reduction of fractions 
based on memorized algorithms. 
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Finally, because of changes in the sample of schools 
over time and the variable amounts of data provided by 
different teachers, a statistical model was fit to estimate 
time trends, adjusting for the missing data. We com- 
puted both an overall composite trend and separate 
trends for challenging versus typical assignments. See 
Appendix B for further details on the statistical model 
used to estimate these trends. 

Trend Results 

The quality of classroom assignments in the field sample 
of Annenberg Challenge schools improved between 1997 
and 1999. As shown in Table 1 and Figures 1 and 2, 
overall scores were generally higher in the two years fol- 
lowing 1997, except in sixth grade math where the scores 
appeared unchanged. The level of authenticity in chal- 
lenging assignments had clearly increased from 1997 to 
1999, while the results for typical assignments appeared 
somewhat more mixed. In general, both the 1998 and 
1999 results exceeded the base year of 1997. While in 
several instances the 1999 average scores were somewhat 
lower than in 1998, an overall trend of improvement 
remains. As expected, the assignments designated by 
teachers as challenging tended to score higher than typi- 
cal assignments. 

While these improvements are encouraging, the over- 
all level of challenge in math assignments still remains 
quite low. More than 80 percent of the sixth and 
eighth grade math assignments in 1 999 provided only 
minimal or no challenge. In contrast, 1999 writing 
assignments showed more evidence of intellectual chal- 
lenge, with 48 percent in sixth grade showing moderate 
or extensive challenge, 56 percent in eighth grade, and 
64 percent of third grade assignments. Details of the 
distribution in the categories for mathematics and writ- 
ing assignments are in Table 2 and Figures 3 and 4. 

The results reported are interim findings from the 
first three years of the Annenberg Challenge. A final 
round of data collection will occur in 2000-200 1 , which 
will provide a more definitive basis for conclusions about 
the extent of school improvement over the last five years 



Table 1 

Average Scores on Intellectual Quality for 
Typical and Challenging Assignments 



Writing Grade 3 





Typical 


Challenging 


Overall 


1997 


3.67 


4.82 


4.28 


1998 


4.34 


4.65 


4.41 


1999 


5.19 


5.29 


5.26 


Writing Grade 6 








Typical 


Challenging 


Overall 


1997 


4.77 


6.53 


5.75 


1998 


7.52 


7.77 


7.61 


1999 


6.19 


6.91 


6.41 


Writing Grade 8 








Typical 


Challenging 


Overall 


1997 


4.55 


6.05 


5.36 


1998 


6.96 


6.78 


6.88 


1999 


5.95 


6.19 


5.97 


Math 


Grade 3 








Typical 


Challenging 


Overall 


1997 


4.34 


5.41 


4.96 


1998 


6.06 


6.37 


6.19 


1999 


5.71 


7.06 


6.16 


Math 


Grade 6 








Typical 


Challenging 


Overall 


1997 


5.92 


6.04 


6.00 


1998 


6.00 


6.38 


6.11 


1999 


4.93 


7.20 


5.80 


Math 


Grade 8 








Typical 


Challenging 


Overall 


1997 


4.90 


5.33 


5.07 


1998 


4.79 


5.86 


5.11 


1999 


5.19 


6.54 


5.63 



Note: The data above are a measure of relative change over 
time within grade level, subject across years, and should not 
be compared across grades or subjects. 
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Figure 1 

Trends in Writing Assignments 

1997 to 1999 





3rd Grade 



6th Grade 



8th Grade 
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Trends in Mathematics Assignments 

1997 to 1999 



Figure 2 




1997 1998 1999 




3rd Grade 



6th Grade 



8th Grade 
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in our sample of Chicago Annenberg schools. Although 
the results presented above indicate some positive trends, 
we think it is premature to draw conclusions from these 
data about effects of either the Chicago Annenberg Chal- 
lenge or CPS policies on the quality of teachers’ assign- 
ments. Future project analyses that take into account 
system-wide test score trends, survey reports, new evi- 
dence on teachers’ assignments and student work, as well 
as in-depth studies of selected schools will provide much 
stronger evidence on the extent of school improvement 
and the factors that affected it. 



Table 2 

Distribution of Classroom Assignments by 
Level of Intellectual Challenge 



1999 Writing 





3rd Grade 


6th Grade 


8th Grade 


None 


6% 


18% 


6% 


Minimal 


31% 


34% 


39% 


Moderate 


53% 


40% 


56% 


Extensive 


11% 


8% 


0% 


1999 Math 




3rd Grade 


6th Grade 


8th Grade 


None 


27% 


65% 


40% 


Minimal 


27% 


18% 


51% 


Moderate 


42% 


10% 


9% 


Extensive 


4% 


7% 


0% 



Note: Numbers may total more than 100 due to rounding. 



Visit the Consortium’s Web site at: 

www.consortium-chicago.org 
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1999 Writing Assignments 



Figure 3 




3rd Grade 



BNone 



6th Grade 8th Grade 

Extent of Challenge 

□ Minimal □ Moderate □ Extensive 



1999 Mathematics Assignments 



Figure 4 




Extent of Challenge 

HNone □Minimal □ Moderate □ Extensive 
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Endnotes 

1 For previous Consortium work on this topic see Bryk, Anthony 
S., Yeow Meng Thum, John Q. Easton, and Stuart Luppescu 
(1998), Academic Productivity of Chicago Public Elementary 
School W, Easton, John Q., Brian Jacob, Stuart Luppescu, and 
Melissa Roderick ( 1 998) , Adjusting Citywide TTBS Scores for Stu- 
dent Retention in Grades Three, Six, and Eight, Easton, John Q., 
Todd Rosenkranz, Anthony S. Bryk, Brian A. Jacob, Stuart 
Luppescu, and Melissa Roderick (2000), Annual CPS Test Trend 
Review, 1999. In terms of recent test score trends for Annenberg 
schools see Smylie, Mark A., Stacy A. Wenzel, Penny Bender 
Sebring, Elaine Allensworth, Tania Gutierrez, Sara Hallman, 
Shazia Rafiullah Miller, with Stuart Luppescu (in progress). Im- 
provement in Chicago Annenberg Schools, 1996-1999: Interim Tech- 
nical Report of the Chicago Annenberg Research Project. 

2 The main public report of this research was Newmann and 
Wehlage (1995) which was distributed by the Wisconsin Center 
for Education Research, the American Federation of Teachers, 
the Association for Supervision and Curriculum Development, 
the National Association of Elementary School Principals, and 
the National Association of Secondary School Principals. A more 
detailed account is available in Newmann and Associates (1996). 
This research has been referenced in numerous professional pub- 
lications and has been the subject of professional development 
for numerous schools and districts in the United States and 
abroad. The standards for intellectual quality have been integrated 
into Michigan’s state curriculum standards and assessments and 
are the focus of research and school development projects in Aus- 
tralia. To be sure, this is not the only conceptual framework with 
associated scoring rubrics that could be applied to assess the in- 
tellectual challenge of classroom assignments. However, the frame- 
work is considered useful in many places for helping teachers 
move beyond traditional teaching of basic skills to more ambi- 
tious intellectual work. 

3 Newmann, Lopez, and Bryk (1998). 

4 See Newmann, Bryk, and Nagaoka (in press) for earlier results 
on this topic. 

5 Since the study began in mid-winter of the 1996-1997 school 
year, the project collected only two challenging and two typical 
assignments for a total of four assignments that year. 

6 Detailed descriptions of the standards and the rubrics used to 
score assignments are available from the Consortium on Chi- 



cago School Research. The standards for construction of knowl- 
edge and connection to students’ lives were scored on a 3 -point 
scale and elaborated communication on a 4-point scale. 

7 Working in grade level teams for each subject (writing and math- 
ematics) and scoring one standard at a time, each assignment 
was initially scored on that standard by one of the teachers in the 
team. Then each year at least 60% of the assignments were scored 
independently (i.e., without knowledge of the initial score) on 
that standard by a different teacher in the team. The double- 
scored assignments were selected randomly from the pool of as- 
signments. Each teacher in the team was paired with every other 
teacher to double score at least one of the standards. 

8 The many-facet Rasch measurement model used here is: log(P nijk 
/P „■.«)= B - D - C - F. where P is the probability of assign- 
ment n being given a rating of k on standard i by judge j, P ( is 
the probability of assignment n being given a rating of k-1 on 
standard i by judge j, B ^is the intellectual challenge of assign- 
ment n, D. is the difficulty of standard i, Cj is the severity of 
judge j, and F k is the difficulty of receiving a score in category k 
rather than a score in the next lower category, k-1. See Linacre 
(1993) for further explanation of the model. Thus the final mea- 
sure of the intellectual challenge in any assignment aggregates 
the score across all three standards, adjusted for the difficulty of 
each standard and the relative severity of the scorers. As in other 
item response theory applications, the final measure of assign- 
ment challenge exists in a logit metric. For reporting purposes 
we converted them to a simple 0 to 10 point scale. 

9 The cross-year rescoring design allowed us to adjust for all the 
1997 and 1998 measures based on the average difference be- 
tween the original scores and the 1999 re-scores. This adjust- 
ment was then added to the scores for all of the 1997 and 1998 
assignments to put them on the same scale as the 1999 assign- 
ments. Specifically, we calculated a Tukey’s bi-weighted mean 
for the difference between the original scores and rescores for 
each year, subject, and grade. This bi-weighted mean was cho- 
sen as the adjustment statistic because it is a robust statistic that 
down weights the influence of extreme outliers in the data. This 
seemed most appropriate, given that our difference statistic in- 
volved a number of extreme values (i.e., the distribution had 
“fat tails”). For a further discussion of this statistic see Mosteller 
andTukey (1977). 
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APPENDIX A 



Additional Statistical Details on Classroom Assignments Data 



Table A1. 1997 Demographics of CPS Schools and 
Chicago Annenberg Challenge Schools 





CPS 


CAC Schools 


Sample Schools 


% At or Above Natl. Avg.- Reading 


29.9% 


28.3% 


24.5% 


% At or Above Natl. Avg.- Math 


37.1% 


34.9% 


31.0% 


% African-American 


58.3% 


59.8% 


53.0% 


% Latino 


28.3% 


29.4% 


38.9% 


% Low Income 


84.7% 


87.6% 


89.4% 



Table A2. Number of Assignments by Type, Subject, Grade, and Year 





Typical 


Writing 

Challenging 


Total 


Typical 


Math 

Challenging 


Total 


Third Grade 
1997 


35 


33 


68 


32 


32 


64 


1998 


135 


61 


196 


142 


64 


206 


1999 


103 


51 


154 


107 


52 


159 


Sixth Grade 
1997 


34 


39 


73 


27 


26 


53 


1998 


100 


46 


146 


107 


51 


158 


1999 


80 


39 


119 


73 


33 


106 


Eighth Grade 

1997 


21 


27 


48 


22 


21 


43 


1998 


86 


41 


127 


84 


36 


120 


1999 


60 


30 


90 


59 


28 


87 



Table A3. Number of Teachers per Grade 





Grade 3 


Grade 6 


Grade 8 


1997 


23 


29 


22 


1998 


39 


39 


38 


1999 


32 


28 


27 
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Chicago Class/ von 1 Demands for Authentic Intellectual Work 



APPENDIX B 



Additional Details on Statistical Model 

Different teachers and schools participated in this study over the course of three years. As a result, we have data 
from some teachers for three years (19 language arts teachers and 21 math teachers), some teachers for two 
years (35 language arts teachers and 33 math teachers), and many for only one year (102 language arts teachers 
and 91 math teachers). In addition, although we had a fixed data collection design, teachers actually provided 
us with a varying number of assignments, and the composition of these assignments in terms of challenging 
versus typical tasks also varied. Moreover, we found considerable variability among assignment scores from the 
same classroom. Given the high degree of internal variability in assignment scores and the limited amount of 
data from some years in some classrooms for our key predictor variable, we needed to develop a special mea- 
surement model in order to extract the maximum information present in these data. 

We estimated the mean trends in the intellectual demands of assignments in writing and math in separate 
analyses using hierarchical linear models (HLM). The actual analytic model used was as follows. Level 1 was a 
measurement model with an intercept and two effects coded dummy variables for the years 1997 and 1999, 
with 1998 as the excluded category. The outcome variable consisted of the assignment measures generated 
from many-facet Rasch measurement (MFRM). All of the elements in the level- 1 model were weighted by the 
inverse of the standard error of the assignment measures. (These are calculated as a by-product of the MFRM 
analysis.) The major function of this level- 1 measurement model is to take into account the unreliability of the 
assignment scores. Formally, the three coefficients produced here, 7t 1Jk , 7t 2jk , and 7r 3jk , can be thought of as 
latent “true score” for assignment j in classroom k. Each of these becomes an outcome variable in the level 2 
model where we have multiple assignments per classroom. 

Level 1 

V + ( 199? ) + ** ( 19 ") + 

where Y jjk = MFRM assignment quality score for mathematics or writing, and £ jjk is now assumed N(0,1) given 
the re-weighting by the standard errors of measurement. 



Level 2 

PlOk + Pui(( Challen S in S) + r .0k 

P 2 ok + Pzik (Challenging) + r 2Qk 
7t 3 J K= P 30 K + P 3 ik (Challenging) + r 3Qk 



Level 3 

Piok = Yioo + y ioi (Grade 6) + y 102 (Grade 8) + u 
P = y 

P 20k = y 200 + y 20 . ( Grade 6 ) + y 202 ( Grade 8 ) 

P 2 lk — Y 2 10 



Ik 



10 



Psok = Y300 + Y301 (Grade 6) + y 302 (Grade 8) 

^ 31 k ~ Y310 



15 



13 



A dummy variable was entered at level-2 to distinguish challenging from typical tasks with its effect fixed and 
grand-mean centered. As a result, the intercept term P ]Qk is the overall mean score, adjusted for differences 
among teachers in the number and types of assignments they submitted. P 2Qk and P 3Qk are the adjusted year 
effects for 1997 and 1999. Finally, at level 3 (i.e., the classroom level), indicator variables for grade 6 and grade 
8 were included in order to estimate the grade-specific effects. 



This report reflects the interpretation of its authors. Although the Consortium assisted in the develop- 
ment of this research, no formal endorsement by its Steering Committee members, their organizations, 
or the Consortium should be assumed. 
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